fix(report): report eval, generate and pipeline use one paired arm summary (#229 P1-3) - #250

The same point as the previous commit (the review of intel#250: an arm mean
over its own scored questions beside a delta over the pairs), at its
other site in `report model-compare`: each By Difficulty row averaged
the context arm over every context-scored question in the bucket, while
the delta beside it covers the pairs only. On a bucket with q1 80/60,
q2 40/40 and q3 100/- the row printed 73.3 | +10.0; it now prints
60.0 | +10.0, the run summary's numbers on the same questions.

- report/model_compare.py: _difficulty_cells takes the context mean and
  the delta from _run_summary, so a bucket follows the overall summary's
  rule: with pairs, the pair means; context arm only, its context mean
  and no delta; both arms scored but no pair, no mean. A bucket with no
  context score still shows the baseline, as before.
- The model-compare docs and the unreleased fragment name the
  difficulty rows; the goldens (the By Difficulty context cell of eight
  sections), the growth baseline of model_compare.py and its count in
  the records guard (one condition-key read fewer) are lowered.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* fix(dashboard, report): average both arms over the complete pairs

Settles the point the review of #250 left for a follow-up: the dashboard
and the `report model-compare` run summary averaged each arm over its own
scored questions, while `report generate`, `report eval` and the pipeline
average both over the questions scored in both arms (paired_summary). On
an eval file with q1 80/60, q2 40/40 and q3 100/- the dashboard and
model-compare printed 73.3 / 50.0 beside a +10.0 delta; `report generate`
prints 60.0 / 50.0 / +10.0.

- dashboard/aggregator.py: the snapshot and each question-source group
  take both arm means and the delta from the pairs (_arm_means). The
  delta is the difference of the pair means, as in paired_summary, not
  the mean of the stored per-question delta.
- report/model_compare.py: _run_summary averages with_avg and
  without_avg over the same pairs as delta_avg.
- In both, a run that scored one arm only keeps that arm's mean (the
  dashboard's doc_score and the baseline-only ranking read it); with
  both arms scored but no pair, neither mean is given.
- The stats module docstring, the unreleased #250 fragment, the
  quickstart and the model-compare docs say so; the model-compare
  goldens and the growth baseline of _load_snapshot are updated.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* fix(report): model-compare difficulty rows average over the pairs

The same point as the previous commit (the review of #250: an arm mean
over its own scored questions beside a delta over the pairs), at its
other site in `report model-compare`: each By Difficulty row averaged
the context arm over every context-scored question in the bucket, while
the delta beside it covers the pairs only. On a bucket with q1 80/60,
q2 40/40 and q3 100/- the row printed 73.3 | +10.0; it now prints
60.0 | +10.0, the run summary's numbers on the same questions.

- report/model_compare.py: _difficulty_cells takes the context mean and
  the delta from _run_summary, so a bucket follows the overall summary's
  rule: with pairs, the pair means; context arm only, its context mean
  and no delta; both arms scored but no pair, no mean. A bucket with no
  context score still shows the baseline, as before.
- The model-compare docs and the unreleased fragment name the
  difficulty rows; the goldens (the By Difficulty context cell of eight
  sections), the growth baseline of model_compare.py and its count in
  the records guard (one condition-key read fewer) are lowered.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

---------

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

The summary averages the questions scored in both arms (paired_summary, intel#250),
so an unjudged retrieval miss is in none of the three numbers. The run result
now carries paired and unpaired, and the printed line names them.

Thread: PRRT_kwDOVBg7g86xgwon
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…ft unused

main's paired_summary (intel#250) replaced the BASELINE/CONTEXT/records reads in
run(); the merge kept this branch's import line, which ruff F401 rejects.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…mmary (intel#229 P1-3) (intel#250)

* fix(report): average both arms over the complete pairs in report eval and generate

report eval and report generate read the same eval file and printed different
arm means and deltas. report eval averaged the arms over the complete pairs;
report generate averaged each arm over every question scored in it, so the
context and baseline means could cover different questions and their
difference was not the delta printed beside them. The pipeline run summary
had the same per-arm averages.

All of them now call eval.stats.paired_summary, which takes the complete pairs
only: the arm means, min/max and delta, and the improved/degraded counts. The
delta is the difference of the paired means, which equals the mean of the
paired deltas. The trust gate and the significance test already used complete
pairs. Unpaired questions are counted (paired_questions, unpaired_questions in
the JSON; a note and the Count cell in the Markdown), so none drops silently.

report eval also rounded each arm mean before subtracting. The delta is now
the unrounded difference rounded once for display, so it can move by 0.1.

Refs intel#229 (P1-3)

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(report): pin the unrounded-mean delta in report eval and generate

The agreement fixture has arm means exact at one decimal, so putting back
the rounded-difference delta in report eval passed every test. Three pairs
with means 80.67/70.33 pin +10.3 in both reports, plus the Breakdown Total
row, the delta min/max and the improvement counts.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

* fix(report): rank the top/bottom performers over the complete pairs

The Overall Max covers only the questions scored in both arms, but the
top and bottom performer tables still ranked every context-arm answer,
so an unpaired answer could score above the Overall Max printed beside it.
Both tables now use the same complete pairs as the Overall statistics,
and the note under the Overall table says so.

The tests now also pin the grouped tables (topic and source): their
means, deltas, unpaired counts and the "(N unpaired)" cell, plus the
without-arm min/max and a delta that is not the difference of rounded
means.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(report): pin the pipeline summary rounding and the count cells

The pipeline summary's delta is the difference of the unrounded means, and
its baseline mean covers the complete pairs only. The eval report's Summary,
breakdown and difficulty rows print the unpaired count beside the question
count. Each of these had a surviving mutant. The new assertions kill them.

Thread: PRRT_kwDOVBg7g86wpmIH
Review: PRR_kwDOVBg7g88AAAABUqK69Q#1

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(stats): say which reports call paired_summary

The docstring said every report that prints arm means or a delta calls
paired_summary. The eval report, the report generator and the pipeline
summary do. The dashboard and the model-compare summary still average each
arm separately, which is follow-up work because it changes their numbers.

Thread: IC_kwDOVBg7g88AAAABgAwfHA.a
Review: PRR_kwDOVBg7g88AAAABUdomSg: already fixed in 68d90f23 and f09eeb77

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(report): divide the win and degradation rates by the complete pairs

The win rate and the degradation rate in Key Observations divided by every
question, but their numerators count only the complete pairs. An unpaired
question therefore lowered both rates. They now divide by the complete pairs,
the same set the Overall delta uses.

Thread: PRR_kwDOVBg7g88AAAABUbZiqA

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog): name the other report numbers the pairing changes

The fragment did not say that the top/bottom performer tables and the
Key Observations win and degradation rates now count only the complete
pairs, nor that the dashboard and the model-compare summary keep the
per-arm means and so differ from report generate on unpaired files.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

---------

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>

…tel#286)

* fix(dashboard, report): average both arms over the complete pairs

Settles the point the review of intel#250 left for a follow-up: the dashboard
and the `report model-compare` run summary averaged each arm over its own
scored questions, while `report generate`, `report eval` and the pipeline
average both over the questions scored in both arms (paired_summary). On
an eval file with q1 80/60, q2 40/40 and q3 100/- the dashboard and
model-compare printed 73.3 / 50.0 beside a +10.0 delta; `report generate`
prints 60.0 / 50.0 / +10.0.

- dashboard/aggregator.py: the snapshot and each question-source group
  take both arm means and the delta from the pairs (_arm_means). The
  delta is the difference of the pair means, as in paired_summary, not
  the mean of the stored per-question delta.
- report/model_compare.py: _run_summary averages with_avg and
  without_avg over the same pairs as delta_avg.
- In both, a run that scored one arm only keeps that arm's mean (the
  dashboard's doc_score and the baseline-only ranking read it); with
  both arms scored but no pair, neither mean is given.
- The stats module docstring, the unreleased intel#250 fragment, the
  quickstart and the model-compare docs say so; the model-compare
  goldens and the growth baseline of _load_snapshot are updated.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* fix(report): model-compare difficulty rows average over the pairs

The same point as the previous commit (the review of intel#250: an arm mean
over its own scored questions beside a delta over the pairs), at its
other site in `report model-compare`: each By Difficulty row averaged
the context arm over every context-scored question in the bucket, while
the delta beside it covers the pairs only. On a bucket with q1 80/60,
q2 40/40 and q3 100/- the row printed 73.3 | +10.0; it now prints
60.0 | +10.0, the run summary's numbers on the same questions.

- report/model_compare.py: _difficulty_cells takes the context mean and
  the delta from _run_summary, so a bucket follows the overall summary's
  rule: with pairs, the pair means; context arm only, its context mean
  and no delta; both arms scored but no pair, no mean. A bucket with no
  context score still shows the baseline, as before.
- The model-compare docs and the unreleased fragment name the
  difficulty rows; the goldens (the By Difficulty context cell of eight
  sections), the growth baseline of model_compare.py and its count in
  the records guard (one condition-key read fewer) are lowered.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

---------

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* fix(answers): Stack P asks the doc source for top_k docs (#228 step 2)

The answerer stored top_k and never sent it, so retrieval asked the doc source for its default max_results. It now sends max_results=top_k, the same request DocTreatment makes on Stack A. With the default top_k of 3 the reranker sees 3 raw docs, not 5, so with-docs answers and their judge scores move.

Flips the PINNED rows D1 retrieval.passed and D1 retrieval.max_results. Test fakes that take the doc-source call now accept max_results.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(benchmark): benchmark run takes and passes --judge-temperature (#228 step 2)

benchmark run had no judge temperature flag, so its judge always ran at the provider default while evaluate and arms run took one. The flag is added (default unset, so runs without it are unchanged) and reaches the EvaluationPipeline judge.

Flips the PINNED row D6 cli.judge_temperature_flag. benchmark batch does not get the flag yet.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(pipeline): resolve the doc library id through the registry (#228)

EvaluationPipeline took the library id from --context7-id or the client's
own resolver, so a product whose products.yaml declares context7_id was
still looked up by its registry key. Resolve it with the shared
resolve_library_id_for_product helper, the same order the arms stack uses:
explicit id, then the registry's context7_id, then the client.

Question generation and answer generation both go through the helper.
The registry is injectable (registry=) and defaults to the bundled
products.yaml. Parity row D8 flips from client/resolved to registry/lib.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(judge): do not score the no-retrieval placeholder by default (#228)

When no document reached the prompt (no hit, or the reranker raised), the
answerer records "[FALLBACK: No relevant documentation found]" as the
with-docs answer, and the judge scored that text like a model answer. It
inflated the with-docs side of the paired delta on questions the docs did
not answer.

The judge now leaves a context answer whose doc_source is a no-retrieval
source (NO_RETRIEVAL_SOURCES) unjudged, so its with-docs score and delta
are None and the question drops out of the paired statistics. The record
and its retrieval_hit stay. Opt back in with judge_placeholder=True, or
--judge-placeholder on benchmark run, orchestrate evaluate and eval score.

The trust gate's retrieval_hit_rate read only the both-arms-scored
evaluations. Unjudged misses would then vanish from it and an all-miss run
would read as "not measured". It now counts every evaluation's
retrieval_hit. The thresholds are unchanged.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* refactor(answerer): delete dead code in the context arm (#228)

Answerer built a self.llm client in __init__ that nothing read: the answer
call goes through llm_call_with_usage, which picks the provider's client
itself. Only the unsupported-provider ValueError is kept; the client
construction and its imports are gone.

_generate_with_docs had an "if not docs" branch that could not run, because
_generate_answer_pair calls it only when retrieval returned documents. The
branch is deleted with its unreachable "[No documentation retrieved]" record.

No parity row changes: nothing observable on either stack moved.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* feat(artifacts): stamp method_version in the answers and judge output (#228)

The Stack P answers artifact and the judge's evaluation output now carry
method_version, from defaults.ANSWERING_METHOD_VERSION (2). The steps
before this one are version 1 and have no stamp, so a later compare can
refuse to put runs of different versions side by side. compare does not
read the stamp yet.

answers.v1 and eval.v1 document the field. The arms artifact is not
stamped in this change; the new parity row records that.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(parity): import List for the paired-delta helper (#228)

ruff F821 flagged the annotation on _paired_deltas. The module uses
`from __future__ import annotations`, so the missing name did not fail at
import time and the parity tests passed without it.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(answerer): one provider allow-list smoke test replaces the init set-then-read tests (#228)

With the llm construction removed from Answerer.__init__, test_init_openai and
test_init_anthropic only set attributes and read them back, which the test
quality audit flags (set_then_read). The new parametrized test checks what
init does now: each supported provider is accepted. It is marked as a
must-not-raise smoke test.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(answerer): pin the top_k value at the Answerer call site (#228)

test_calls_mcp_client builds the Answerer with the default top_k, so a literal
max_results=5 would pass it. The new test builds it with top_k=3 and asserts
the doc source is asked for 3. On intel/main the same probe sends no
max_results at all, so the test fails there.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(gate): say which trust-gate checks run on both-arm-scored questions (#228)

The methodology list and the method_version comment described the trust gate
and the stamp more broadly than the code does. The score checks run over the
both-arm-scored questions, so a question with no document leaves them. Nothing
reads method_version yet; the comment now says so.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(gate): count unjudged retrieval misses as unusable in zero_score_fraction (#228)

A question whose retrieval missed is no longer judged with docs, so it has
no context score. The zero-score check counted only scored questions, so a
run where most questions miss passed it: 4 misses of 14 gave 0.0 PASS, where
intel/main gave 0.286 FAIL (the misses were judged at zero there).

The check now counts a miss (no context score, a baseline score, and
retrieval_hit False) as unusable, in the numerator and the denominator. The
message says such rows had no document. Thresholds, min_questions and the
early return are unchanged. The new fail-severity retrieval check is not
added here.

Regression test: tests/test_trust_gate_unjudged_misses.py. It fails on the
parent commit (4 of 14 passes at 0.0) and passes with this change.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(gate): zero-score bullet counts retrieval misses; parse evaluations once (#228)

The methodology bullet still said the zero-score fraction leaves out
questions with no document. It counts them now (c517c72). Update it.

Also parse the evaluations once in evaluate_trust. The input may be a
one-shot iterable, and the old code parsed it again for the retrieval check.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(judge): a judge-error row keeps the retrieval outcome of its answer

The retrieval coverage floor counted every judge-error row as an unrecorded
outcome, so a run with more than 20% judge errors lost the retrieval check.

Thread: PRRT_kwDOVBg7g86xgwoY
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(diagnoser): an unjudged retrieval miss is an empty retrieval

With the no-retrieval placeholder left unjudged its delta is None, and
diagnose() returned insufficient_data before it looked at the doc source.

Thread: PRRT_kwDOVBg7g86xgwog
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(pipeline): say how many questions the run summary left out

The summary averages the questions scored in both arms (paired_summary, #250),
so an unjudged retrieval miss is in none of the three numbers. The run result
now carries paired and unpaired, and the printed line names them.

Thread: PRRT_kwDOVBg7g86xgwon
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(benchmark): resolve the library id through the --registry that was loaded

benchmark run and batch loaded the registry for the entry, but the pipeline
looked the key up in the bundled products.yaml. A custom entry without a
context7_id then reached Context7 as a bare key and retrieved nothing, and a
custom key that the bundled registry also has took the bundled id.

Thread: PRRT_kwDOVBg7g86xgwou
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(pipeline): pin the explicit id and the registry-key lookup of _library_id

Both mutants named in the review (explicit_id=None, product instead of
product_key) now fail a test.

Thread: PRRT_kwDOVBg7g86xiJYO
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(judge): stamp whether the placeholder was judged in the eval output

--judge-placeholder gives different numbers on the same answers under the same
method_version. The eval output now carries judge_placeholder (declared in
eval.v1), so two files of one version that are not comparable can be told apart.

Thread: PRRT_kwDOVBg7g86xgwoz
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(gate): show the retrieval rate when no question is scored in both arms

With every question an unjudged miss the gate returned after min_questions
with "0 questions evaluated", pointing away from the cause. The retrieval
check is now a helper and also runs on that path. The 0-hit case has its
test again, and the 2-hit test is named for what it checks.

Thread: PRRT_kwDOVBg7g86xgwo7
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(gate): pin the three conditions of the unjudged-miss term

A legacy row, a hit that was not scored and a miss without a baseline must
leave zero_score_fraction unchanged; each mutant of the filter named in the
reviews now fails. The judge stub branch that only a judged miss reached is
gone: the default leaves misses unjudged.

Thread: PRRT_kwDOVBg7g86xiJYM
Thread: PRRT_kwDOVBg7g86xlsnp
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(cli): drive the real handlers for --judge-temperature and --judge-placeholder

benchmark run (single and multi-run) is driven through cmd_benchmark_run with
a faked pipeline; evaluate and eval score run end to end against a mocked
judge transport. Each removed call-site kwarg named in the reviews now fails a
test. The old test named for the handler called _run_single_library directly.

Thread: PRRT_kwDOVBg7g86xiJYT
Thread: PRRT_kwDOVBg7g86xlsUD
Thread: PRRT_kwDOVBg7g86xlsb0
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog): name the flags and commands that exist; say old scores differ

benchmark run has no --top-k and there is no orchestrate evaluate: the command
is dokimi evaluate. The fragment now says what each command asks for, that
answers generate changes too, that old scores are not comparable, and covers
the fixes of this round.

Thread: PRRT_kwDOVBg7g86xgwo3
Thread: PRRT_kwDOVBg7g86xlsKV
Thread: PRR_kwDOVBg7g88AAAABU4c24A
Thread: PRR_kwDOVBg7g88AAAABU5LNng
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(answerer): drop the ChatOpenAI patch the module no longer uses

dokimi.eval.answerer neither imports nor uses ChatOpenAI; the patch with
create=True and its mock did nothing and suggested the init builds a client.

Thread: IC_kwDOVBg7g88AAAABg14M_w
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* chore(pipeline): drop the records imports that the merge with main left unused

main's paired_summary (#250) replaced the BASELINE/CONTEXT/records reads in
run(); the merge kept this branch's import line, which ruff F401 rejects.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(pipeline): pin the library id the question step hands the topic extractor

Codecov flagged the one patch line in _generate_questions that no test reached.
The test asserts the extractor gets the registry's declared id; reverting the
line to the client's name lookup fails it.

Thread: IC_kwDOVBg7g88AAAABgqSv4w
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test: pin the branches the #274 review nits left untested

Three nits from the review of #274, one commit:

- test_diagnoser: an unjudged `none` retrieval miss is an empty retrieval,
  and an errored placeholder answer stays insufficient_data. Deleting the
  `not error` guard now fails a test.
- test_pipeline_library_id: a product key that is not in the registry is
  resolved by the client under the key. Passing the display name instead
  now fails a test.
- test_pipeline_summary: an all-paired run does not print "left out".
  Printing the suffix unconditionally now fails a test.

No source change: each new test was checked against the mutation the nit
describes (red), then the unchanged tree (green).

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog): say benchmark run --judge-temperature is a new flag

The flag did not exist before this PR: benchmark run rejected it as an
unknown argument. The entry moves it from Changed and Fixed to Added, so
the Fixed section no longer repeats the claim.

Thread: PRRT_kwDOVBg7g86xtZgo
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog): say a judge-error row on a retrieval miss counts in zero_score_fraction

A judge-error row whose retrieval missed now counts as unusable in the
zero-score check; one on a retrieval hit is still left out. The fragment
did not say so, and the check is fail-severity.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs: keep the comment blocks this PR adds to two lines

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(gate): a judge error on a retrieval miss counts as unusable

The zero-score check counted an unjudged retrieval miss, but a judge error
on a miss has no scores either and was left out. The error row is still a
miss, so it is counted; a judge error on a hit stays out, as before. Told
apart by the record's error field, not by its score shape.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(answerer): a missing top_k asks the doc source for the default

Answerer passed max_results=self.top_k to the doc source, so a None top_k
reached the client as max_results=None, which returns every chunk. Make the
Answerer default and an explicit None both DEFAULT_RETRIEVAL_TOP_K (3). The
answers command already resolves --top-k to that default before it gets here;
this closes the direct-construction path.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(pipeline): document registry and judge_placeholder in Args

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* docs(changelog): limit what --judge-placeholder restores

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* docs(methodology): zero-score bullet counts judge-error misses

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* style(answerer): one import from dokimi.defaults

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(trust-gate): pin the scored-miss clause of the unjudged-miss filter

A miss the judge scored with a 0 context score is already a zero; a
mutant that drops the score-is-None clause counts it twice and passes
the existing tests. The new case asserts 1/11, and fails on that mutant.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(answerer): pin a non-default top_k so an override cannot be ignored

The explicit top_k case used 3, which is also the default. With the
constructor ignoring the caller, the suite still passed. The case now
uses 7 and expects max_results=7; the mutant fails it.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(methodology): name the both-arm subset in the three metric bullets

The zero-score fraction uses an expanded subset; the minimum with_docs
average, the average delta and the coefficient of variation do not.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(cli): allowlist --judge-placeholder, which this stack adds

No registry concept covers the placeholder judge yet, so the spelling
gets a reasoned allowlist row, as the contract test asks.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(cli): regenerate the CLI snapshot after the merge of main and this stack's flags

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(eval): eval score stamps the method version of the answers it judges

An answers file with no method_version is legacy (version 1). Judging it used to stamp the eval artifact with the current version, so legacy answers re-judged today read as version 2 next to answers generated under the new method. The Judge now takes the answers' version; answers generated in process keep the current one.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(pipeline): an mcp source's ;id= outranks the registry's context7_id

resolve_library_id_for_product checks the registry before the client, so a registered product replaced the id named in an mcp: descriptor. The pipeline now passes the descriptor's id as the explicit id, after the --context7-id option and before the registry, as it did before the registry lookup.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog,methodology): say what the stamp and the zero-score bullet cover

Legacy artifacts are version 1 but readers do not normalize that yet; the zero-score bullet names the case where no question is scored in both arms.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(pipeline): eval judges answers with the method version of their artifact

The pipeline built its Judge without answers_method_version, so answers
read from a version 1 artifact, or from an unstamped legacy file, were
stamped as version 2. The version now comes from the artifact it read;
fresh in-memory answers keep the current version.

Thread: PRRT_kwDOVBg7g86yJ-x3
Comment: IC_kwDOVBg7g88AAAABhDPhBg
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* test(trust-gate): expect the zero-score wording the gate now uses

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* Address review: a judged placeholder with no baseline is insufficient data

With judge_placeholder the context answer is scored, so a None delta
comes from a missing baseline. diagnose() classed any no-retrieval
context answer with no delta as an empty retrieval; it now does so only
when the context was left unjudged.

Thread: PRRT_kwDOVBg7g86yMEiW
Thread: IC_kwDOVBg7g88AAAABhDPhBg
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* Address review summaries: points already fixed on this branch

Each review below raised points that its inline threads cover; the fixes
are in the commits that answer those threads (the gate clauses are pinned
by test_trust_gate.py and test_trust_gate_unjudged_misses.py).

Thread: PRR_kwDOVBg7g88AAAABU4c24A: already fixed in the inline threads' commits
Thread: PRR_kwDOVBg7g88AAAABU5LNng: already fixed in the inline threads' commits
Thread: PRR_kwDOVBg7g88AAAABVF59Bw: already fixed in the inline threads' commits
Thread: PRR_kwDOVBg7g88AAAABVGGmvw: already fixed in 5fae382, 6dc1f42 and 805aa58
Thread: PRR_kwDOVBg7g88AAAABVHLzTg: already fixed in the inline threads' commits
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* Refuse a CLI judge's temperature in benchmark run before any answer

The judge refuses a temperature for claude-cli, but only after
benchmark run had generated every answer. benchmark run and batch
now run the same check first, exit 2, as dokimi run does.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* Reword the unjudged-miss comment in diagnose()

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(gate): keep stored judge-error rows out of the retrieval coverage count

Eval files written before error rows carried retrieval_hit turned a measured
hit rate into "Not measured" (and a failing rate into a pass) once the
denominator became every evaluation. Such a row now counts as neither
measured nor unrecorded, as before; new error rows still count.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

* fix(answers): refuse a top_k below 1 now that it reaches the doc source

top_k is sent as max_results, so answers generate --top-k 0 (or a config
retrieval.top_k of 0, "3" or true) made every answer an unjudged placeholder
and still exited 0. The flag takes positive_int, as dokimi evaluate does, and
a bad config value exits 2 before any question is answered.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

* fix(eval): refuse a malformed answers method_version before judging

eval score and the pipeline copied the answers' method_version into the eval
unchecked: "2", true or 3.5 failed eval validation after judge calls were
paid for and left no file, 0 became 1 and -7 went through. A stamp that is
not a positive int now exits 2 (eval score) before any judge call.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

* fix(pipeline): key the topics cache on the doc source too

The library id now comes from the registry for local:, url: and okf:
sources as well, and .cache/topics was keyed on that id alone, so a second
doc set of one registered product silently got the first set's topics.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

* fix(dashboard): score a no-error retrieval miss as context 0

A question the context arm never answered, with no retrieval hit and a baseline score,
counted as no score, so the snapshot averaged the baseline alone (0 hits gave doc_score
equal to the baseline). It now counts as 0.0 for the context arm.

Thread: IC_kwDOVBg7g88AAAABhdSHJQ.a
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(report): say when there are no complete pairs

With no question scored in both arms, report eval printed a 0.0 arm mean, an
Overall delta of +0.0 and a delta over 0 pairs. The Summary cells and the Overall delta
line now show an em dash and a note that there are no complete pairs.

Thread: IC_kwDOVBg7g88AAAABhdSHJQ.b
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog): an absent or null method_version is version 1

The changelog said a non-positive-integer stamp is refused; a null or absent one is the
legacy version 1 and is accepted.

Thread: PRRT_kwDOVBg7g86yPRma
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(methodology): qualify the trust-gate question and denominator rules

The minimum question count excludes an unjudged question only for the single Judge; a
panel judges a fallback-only answer. The zero-score denominator adds retrieval misses with
an unscored context arm and either a scored baseline or a judge error, and no other miss.

Thread: PRRT_kwDOVBg7g86yPRmb
Thread: PRRT_kwDOVBg7g86yPRmc
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(pr): PR body names the legacy stamp and the 14-of-14 retrieval row

No code change. The body edit (line 22: method_version 1 for an eval of legacy answers;
the 14-of-14 row: retrieval_hit_rate 0.0 FAIL) is applied to the PR description.

Thread: IC_kwDOVBg7g88AAAABhdSHJQ.c
Thread: IC_kwDOVBg7g88AAAABhdSHJQ.d
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(report): the unpaired note says there are no complete pairs

With no complete pair, the note under the Summary still said the means were over 0
complete pairs. It now says there are no complete pairs and that no arm mean or delta is
given.

Thread: IC_kwDOVBg7g88AAAABhdSHJQ.b
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* refactor(dashboard, report): extract helpers to keep the complexity ratchet

The miss-as-zero rule and the no-pairs lines pushed _load_snapshot over the
cap of 10 and generate_report past its baseline of 18. Move them into
_context_score, _overall_line and _delta_summary; no behavior change.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(dashboard): leave an unjudged miss out of the arm means and delta

A no-error retrieval miss scores 0 as the context arm, so the doc score
counts it, but it is no judged measurement: pairing it with its baseline
made the dashboard report a -20 delta where the eval report gave +20 over
the same file. The arm means and delta are now over the measured pairs;
the doc health (doc_score, status) still counts the miss as 0.

Thread: PRRT_kwDOVBg7g86yUBEO
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(report): print no delta for a set with no complete pair

A set with no complete pair printed 0.0 for its context, baseline and delta
in the Dynamic vs Static and difficulty tables, and the agreement bullet
claimed consistency. Those rows now print an em dash like the Summary, and
the agreement bullet says there is no delta when either set has no pair.

Thread: PRRT_kwDOVBg7g86yUBES
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* fix(report): no zero-mean delta or conclusion for an all-miss run

A run with no complete baseline/skill pair printed zero means, a zero
delta and a conclusion. Stats now carry None without a pair; the
overall, cluster and source rows show an em dash, the advice is
suppressed, and the unpaired note explains that no pairs were made.
Also lower the _format_markdown growth baseline to its new complexity.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* Test that Judge diagnoses a judged placeholder without a baseline

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* Name the report changes in the Stack P step 2 changelog

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* test(report): pin the no-pair and miss guards the review found untested

- A group with no complete pair has no means in the grouped tables (topic and source), in JSON or markdown.
- A retrieval miss counts as a context zero only with a clean baseline score: no baseline score, a top-level error and a with_docs error each leave the doc score alone.
- methodology: the single Judge skips the no-retrieval placeholder unless --judge-placeholder is set; JudgePanel judges nonempty fallback answers.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

* docs(changelog): qualify the miss-as-zero and stamping claims

The doc score counts a miss as 0 only with a baseline score and no error
on either arm, and panel-score artifacts carry neither method_version nor
judge_placeholder. Say so where the entry claims otherwise.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

---------

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Co-authored-by: Anton Volkov <antonwolfy@gmail.com>
Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>